RP-11388

Lucene has made the max vector dimensions a property of the codec. We override the function to set it to 4096.

There are some risks with bigger vectors - there is a lot of discussion in apache/lucene#11507. These are the most relevant points from the issue:

1. Bigger vectors require lot more memory.
2. Each added dimension makes the floating point errors larger and sometimes also returns NaN.
3. Lucene's default limit is still 1024 - they can still make changes in the future keeping this in mind.

I thought about not having a limit to allow for easy experimentation, but the above risks are significant. I also considered keeping the limit and make it configurable, but I fear it would effectively be no limit - when needed users will increase it.
I think it's reasonable to still have a limit but increase it to 4096 (same as Elasticsearch). This should allow large enough vector sizes required for experimentation today and not have too much of a risk of performance/correctness/future changes.